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OO ' We derive an exact representation of the topological effect on the dynamics of sequence processing 

' neural networks within signal-to-noise analysis. A new network structure parameter, loopiness coeffi- 

, cient, is introduced to quantitatively study the loop effect on network dynamics. The large loopiness 

04 ■ coefficient means the large probability of finding loops in the networks. We develop the recursive 

equations for the overlap parameters of neural networks in the term of the loopiness. It was found 
I that the large loopiness increases the correlations among the network states at different times, and 

eventually it reduces the performance of neural networks. The theory is applied to several network 
topological structures, including fully-connected, densely-connected random, densely-connected reg- 
ular, and densely-connected small-world, where encouraging results are obtained. 

PACS numbers: 87.10.-l-e, 89.75.Fb, 87.18.Sn, 02.50.-r 

P5 I Starting from pioneering works of modelling complex networks [l|, Q i research area related with complex networks 
• have been growing very fast. In parallel to the studies of structural properties of complex networks, there has also been 
a growing interest in dynamic systems defined on networks. For example, synchronization and collective dynamics, 
epidemic spreading, cascading failures, opinion formation as well as various strategic games, and some physical model 
like Ising model (see ^] and refs. therein). 

Neural assemblies (i.e. local networks of neurons transiently linked by selective interactions) are considered to 
be largely distributed and linked to form a web-like structure of the brain. Many researches suggested that neural 
connectivity is far more complex than the random graph. The cortical neural networks of Caenorhabditis elegans and 
^ cat were reported to be small- world (SW) and scale- free (SF), respectively |l, 4]. It is very important to understand 
O how the complex neural wiring architecture is related to brain functions. With same average connections, Hopfield 
network with random topology was reported to be more efficient for storage and recognition of patterns than both 
SW network and regular network For the SF connection, with the same number of synapses, Torres et al. found 

: that the capacity is larger than the storage of the highly diluted random Hopfield networks [7| . Using a Mente-Carlo 
K>" ' method to lower the clustering coefficient smoothly with the degree of each neuron kept unchanged, Kim found that 
1^ the networks with the lower clustering exhibit much better performance l§\ . 

~l It is well known that the equilibrium properties of fully-connected Hopfield networks had been extensively studied 

] using spin-glass theory, especially the replica method [1, [TO, [HI • And the dynamics of fully-connected Hopfield model 
' with static patterns and sequence patterns were widely studied using generating functional analysis [l2, • As a 
simple relaxation of biological unreaiisticai fully-connected model, various random diluted model were studied, include 
. extreme diluted model [ij, [lB| , finite diluted model [H, [O] , and finite connection model [ll] . However, in the case 
■ of complex network topology, as far as we know, there are hardly any theoretical studies for dynamics or statics. 

In this paper, we use signal-to-noise analysis to study the effect of topology on transient dynamics of sequence 
^ ' processing neural networks. Considering the mathematical convenience, here we only focus on the sequence processing 
] models and the relationship between our results and Hopfield models will be discussed. 

The topological effect on neural networks mainly come from loops of topology [l^ . In the case of so-called extremely 
Q ' diluted structure (linijv-^oo — limTv^oo k/N — 0), average loop length is very big, like logj._]^ N, so number of short 
Q . loops (like triangles, quadrangles) in networks will be very small and effect caused by short loops can be neglected. 
Dynamics of networks in this case is easy to study because each spin at different time steps is uncorrelated [ij] . In 
contrast, if such small loops exist, the correlations and feedbacks in network will lead to more complicated dynamics. 
' In order to quantitatively present the effect from loops, we define a new parameter, loopiness coefficient, to present 
H I the probability of finding loops in the network. The definition is shown in Fig. [TJ In the left triangle, spin j is 
- connected to spin «, and spin k is connected to j. We define the probability that k is connected to i as the first-order 
loopiness coefficient, Li. Similarly, the nth-order loopiness coefficient, L„, denotes the linking probability between 
two vertices to form a loop with n + 2 edges. 

We now study a sequence processing model consisting of N oo Ising- type neurons (t) G {+1, — 1}. The neurons 
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FIG. 1: Left: the first-order loopiness coefficient, Li, means the probability of connectivity between k and i when j is connected 
to i and k is connected to j. Middle: the second-order loopiness coefficient L2. Right: the third-order loopiness coefficient L3. 



update their states simultaneously, with the following probabilities, 



Prob[s, {t+l)\h, (t)] = 



g/3s.(t+l)hi(t) 
2 cosh {Ph^ (t))' 



(1) 



where the local field hi (t) — X^jLi Jij^j (Oj ^^'^ P the inverse temperature. For the transfer function g{ ), we 
denote by Si[t+1) = g{hi (t)). 

Let us store p — aN random patterns S,^ = (<^f 
matrix Jy = Ci^^Cj is chosen to retrieve the patterns as ^ ^ • ■ • ^ — > ^-'^ (note that = ^^), 

where is the adjacency matrix (c^ = 1 if j is connected to i, Cij = otherwise) (20| . Consequently, the degree of 
spin i is fci = S^gt '^ij ~ '^j '-y ' where Ti is the set of spin j connected with i. In this work, for studying the role of 
loopiness, we only consider ki ^ k = Nc. We assume that this property holds for dense connected random network, 
dense connected SW network, and dense connected regular network. 

For any pattern the order parameter is m" (t) = jj'^ (0 which represent the overlap between s (t) and the 
condensed pattern The local field in neuron i is described by 



, in network, where a is the loading ratio. So, the interaction 



h, {t) 



Nc ^ 



(2) 



where the first term of the left-hand side is the signal term and the second one the crosstalk noise from uncondensed 
patterns. 



^» (^) = ]^ E E er^^e;^. . 



(3) 



The state of the spins Si{t) is determined by the sign of local field hi{t), that is. 



Expanding Sj (t) into first order, we obtain 



s, (t) = g {h, (t - 1)) = J ^ ^ ejer'^fc {t 1) 

\ keT, 

+ ]^E^;^r^^(^-i) + ]^ E E?]^erV-(^-i)) 



(4) 



-I- 0{N-^) 



\_ dg_ 

Nc dh 



(5) 
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where 



(t) = 9(^,1: ^^'^^ - 1) + ]^ E E mr'-^^ (t - 1) 

= 9 (^h, - 1) - i- ^ ^^e,-'s, {t - 1) j . (6) 
Note that, in here, (t) is independent of Then the crosstalk noise can be presented by 

^«w = E]^E^repi'w 

+ ^ - 1) ]^ E ]^ E E ^r^r^^ it-i), (7) 

where U{t — 1) = (((ff U(t-i))h,))- {■)h denotes the average over the distribution of the local field, and ((•)) means the 
average over the initial conditions and the condensed pattern. 

Using the central limit theorem, the first term of Eq. ([7]) converge to a zero mean Gaussian form A/'(0, a/c). Since 
Cf^^Cfe~^^fc ~ 1) is not independent, we cannot apply this theorem directly into the second term of Eq. ([5]), we 
have to expand Zi (t) into time t — 2, 



it) = AA(o, a/c) + u{t-i) — Y.w,T.Il ^er'C"' it - 1) 

+U{t-l)U{t^2)^^Y.}-,i:wXi: ^r^'^^ - 2) , (8) 
jeT, keTj fi^v leTk 

with U{t — 2) = (((f||/i(t-2)}h))- Even though averaging over j e Ti, not all k e Tj are residual spins. Note that only 
k belonging to both Ti and Tj will survive the averaging operations. The probability of k is described by the first 
order loopiness coefficient Li (See Fig. [T|). Similarly, from the central limit theorem, it was found that the second 
term of Eq. ([7]) converged to Af{0, LiU'^ (t — 1) a/c). Then we expand Zi (t) to time t — 3, 



Z,{t) ^ U{0,a/c) +M{0,LiU^ {t-l)a/c) 



Nc ^ Nc ^ Nc 

+Uit-l)Uit-2)Uit^3)l-j:Wc^Wc^Wc^T. ^r^a-'^n - 3) . (9) 

JST, keTj leTk neTi n^u 

Here, it is easy to show that the third term of Eq. ^ converge to A/'(0, £2?/^ {t — 1) [/^ (t — 2) a/c). 

Repeating the above expansion up to time 0, we collect all the correlations caused by the loops and derive the final 
expression of crosstalk noise at time t, 

Zdt)=M{<d,a/c) + Y,N{^, U^it[)^ • 

Because each sum in the above equation is independent, one can calculate the variances directly (for proof of inde- 
pendency of each sum, see f2TI. l22j). 
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FIG. 2: (Color online) (a) Temporal evolution of the overlap parameters of network with random topology, the initial overlap 
ranges from 1.0 to 0.1 (top to bottom), (b) basin of attraction of network with random topology, the lower part means the 
lowest initial overlap to retrieve successfully. The upper part is the final overlap for successful retrieval. Temperature is zero. 
The parameters are (a) iV = 5000, c = 0.16, and aN = 190; (b) iV = 3000 and c = 0.1. 



And we write down explicitly the following closed equations of the order parameter m{t), 

m{t + l)= f Dz {^g (^m (t) + a (t) z))^ , 



(12) 



+ 1) = ^ y (5 {^^ it) + <y it) ^))i . (13) 

where (•)^ stands for the average over the retrieval patterns ^, and Dz — exp (— dz. 

We are now ready to present the signal-to-noise treatment of the sequence processing neural networks with narrow- 
degree-distributed topology at zero temperature. The above equations form a recursive scheme to calculate the 
dynamical properties of the systems with an arbitrary time step. However, the following practical trouble is how to 
calculate the loopiness coefficients up to arbitrary order for a given topology. 

In the special case of fully-connected networks, c = 1, all the loopiness coefficient are L„ = 1. And substituting 
into Eq. pT|). we find under the form, 

a^it) = a + U^it-l)(7^{t-l). (14) 

This equation, together with Eqs. (|12j|13p . is coincident with the dynamical equations in [2l|,[2^. The stationary 
equations and simulation results can be found in 21]. 

For random networks, each node has a probability c to connect with others. So the loopiness coefficients are also 
L„ = c. The temporal evolutions of overlap up to 20 time steps obtained by both theory and numerical simulations 
are plotted in Fig. O The initial overlap ranges from 0.1 to 1.0. Fig. [2] shows the basin of attraction from theory and 
simulations. 

However, for regular networks or SW networks, it is impossible to get the exact analytical expressions of the 
loopiness coefficients, so the only solution is programming. Due to algorithm complexity in large size networks, we 
only present the first, second, and third order loopiness coefficients. As a result, one only derives the 4 time step 
dynamics from Eqs. (jlljllSp . In Fig. [31 we show the theoretical results and simulations of 4 time step dynamics for 
regular topology and SW topology with rewiring probability p = 0.5. 

In practice, it was found that the loopiness coefficients decrease with increasing the rewiring probability of SW 
networks, and the overlaps are increaseed with decreasing the loopiness coefficients (see Fig. 0]). From Eq. (|lip . the 
larger loopiness coefficients induce larger variances of the crosstalk noise, and eventually result in a negative effect on 
performances of neural networks. 

Comparing to the sequence processing model, besides of a noise part and a Gaussian part, the local field of Hopfield 
model has a discrete self-interaction part, and crosstalk noise is no longer normally distributed, which produce studied 
intractable. But similarly to sequence processing model, the large loopiness coefficient also increases correlations of 
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FIG. 3: (Color online) Temporal evolution of the overlap parameters with (a) regular topology and (b) SW topology, the initial 
overlap ranges from 1.0 to 0.1 (up to down). Temperature is zero. The parameters are A'^ = 5000, k — 800, and qA'^ — 190. 
Rewiring probability of SW network is 0.5. In our paper, both regular topology and SW topology is in one-dimensional. 
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FIG. 4: (Color online) (a) The first-, second-, and third- order loopiness coefficients and (b) the theoretical results of 4 time 
step dynamics for regular networks, SW networks(p is rewiring probability), and random networks. The initial overlaps are 
0.2. The parameters of networks are = 5000, k = 800, and aN = 190. 



spin states at different time and leads to large variance of the Gaussian part of crosstalk noise. So it is easy to 
qualitatively understand why small amount of randomly rewired connections greatly improve the performance of 
Hopfield model with regular architecture 0, , and why Hopfield model with random networks is better than 

SW networks [(^. 

Note that the value of the first-order loopiness coefficient in undirected graph is the number of triples divided 
by the number of connected triples (See Fig. [1]), which is the same with well-known network parameter clustering 
coefficient Considering Eq. (fTTj) . it shows that Li plays a much more important role than the higher order 
loopiness coefficients, and the lower Li or clustering coefficient make better performance for neural networks. This is 
indeed confirmed by Kim with Monte-Carlo method to tune the clustering coefficient while retaining the same degree 
distributions @]. 

We also have to point out that all topologies studied in both theory and simulations are densely-connected networks, 
our results can not be extended to finite-connectivity networks directly, the reason is: in Eq.(7), using central limit 
theorem, we assume that the term J2fj.^i^ Wc ^j&Ti ^f"*" ^j^j (^) converges to a zero mean Gaussian form A/'(0, a/c). 
This assumption is correct in densely-connected networks where k — > oo, but incorrect in finite-connectivity networks. 
However, we find that if k is big(A: >= 50), above assumption is a good approximation and our theory gives a good 
approximation to finite-connectivity network dynamics. 

At last, it should be very careful that this theory is not suitable for scale- free networks since the power-law degree 
distribution is out of action of signal-to-noise analysis and its loopiness coefficients are degree-correlated, which cannot 
be expressed simply as L„. Attempts of studying dynamics of sequence processing neural networks with power-law 
and even arbitrary degree distributions have been done by neglecting the loopiness effect [27,] . Some numerical studies 
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of properties of neural networks with scale-free topology can be found in [28l |. 

Acknowledgments 

This work was supported by the National Natural Science Foundation of China under Grant No. 10305005 and by 
the Special Fund for Doctor Programs at Lanzhou University. 



D. J. Watts and S. H. Strogatz, Nature(London) 393, 440(1998). 

A. L. Barabasi and R. Albert, Science 286, 509(1999); A. L. Barabasi, R. Albert, and H. Jeong, Physica A 272, 173(1999). 
S. Boccalettia et al, Phys. Rep. 424, 175(2006). 

K. E. Stephan, L. Kamper, A. Bozkurt, G. A. P. C. Burns, M. P. Young, and R. Koter, Phil. Trans. R. Soc. Lond. B Biol. 

Sci. 3561159(2001); C. Cherniak, J. Neurosci. 14, 2418(1994); J. W. Scannell, G. A. P. C. Burns, C. C. Hilgetag, M. A. 

ONeill, and M. P. Young, Cerebral Cortex 9, 277(1999). 

J. J. Hopfield, Proc. Nat. Acad. Sci. USA 79, 2554(1982). 

P. N. McGraw and M. Menzinger, Phys. Rev. E 68, 047102(2003). 

J. J. Torres, M. A. Munoz, J. Marro, and P. L. Garrido, Neurocomputing 58, 229(2004). 

B. J. Kim, Phys. Rev. E 69, 045101(2004). 

M. Mezard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond (World Scientific, Singapore, 1987). 
D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. Rev. A 32, 1007(1985). 

D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. Rev. Lett. 55, 1530(1985). 
A. C. C. Coolen, arXiv:cond-mat/0006011 

A. Diiring, A. C. C. Coolen, and D. Sherrington, J. Phys. A 31, 8607(1998). 

B. Derrida, E. Gardner, and A. Zippelius, Europhys. Lett. 4, 167(1987). 
T. L. H. Watkin and D. Sherrington, J. Phys. A 24, 5427(1991). 

P. Zhang and Y. Chen, Lect. Notes Comput. Sci. 4491, 1144(2007). | |cond-mat/0611313[ | 
W. K. Theumann, Physica A 328, 1(2003). 

B. Wemmenhovel and A. C. C. Coolen, J. Phys. A 36, 9617(2003). 

E. Barkai, I. Kanter, and H. Sompolinsky, Phys. Rev. A 41,590(1990). 
H. Sompolinsky and L Kanter, Phys. Rev. Lett. 57, 2861(1986). 

M. Kawamura and M. Okada, J. Phys. A 35, 253(2002). 

K. Kitano and T. Aoyagi, J. Phys. A 31, L613(1998). 

S. Amari, Proc. IEEE Conference on Neural Networks 1, 633(1988). 

D. BoUe, G. Jongen, and G. M. Shim, J. Stat. Phys. 91, 125(1998). 

O. K. Dekhtyarenko, Proc. IJCNN'05 2, 1178(2005). 

L. G. MoreUi, G. Abramson, and M. N. Kuperman, Eur. Phys. J. B 38,495(2004). 

P. Zhang and Y. Chen, Physica A 387, 1009(2008); Y. Chen, P. Zhang, L.C. Yu, and S.L. Zhang, larXiv:0705.3679V l. 
(2007). 

D. Stauffer, A. Aharony, L. da F. Costa, and J. Adler, Euro. Phys. J. B 32, 395(2003). 



